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Introduction 


Select  two  teams  whose  individuals  are  relatively  equal  in  experience,  training,  skills, 
and  background.  Provide  these  teams  with  the  same  training  and  then  observe  them  perform 
in  their  natural  environment.  Often  one  team  will  perform  better  than  the  other.  Why  do 
these  two  comparable  teams  perform  differently?  Why  is  one  more  effective?  These  are  the 
questions  the  Army  posed  to  Klein  Associates  Inc.  Answering  these  questions  is  the  central 
focus  of  this  research  project. 

The  Army  noticed  that  though  various  units  would  go  through  similar  training  regimes 
they  might  have  radically  different  performance  levels  during  their  rotations  at  the  combat 
training  centers  (CTCs).  This  became  even  more  evident  as  the  Army  began  efforts  on  their 
Home  Station  Determinants  Project.  The  concept  behind  home  station  determinants  was  to 
look  at  the  relationships  between  aspects  of  a  unit’s  home  station  training  (content,  sequence, 
methods,  resources  expended)  and  its  CTC  performance.  These  training  determinants 
encompass  the  unit’s  collective  training  as  well  as  the  individual  soldier  training  issues. 

What  are  conspicuously  absent  are  the  dynamics  of  how  team  members  can  interact  most 
effectively  to  accomplish  their  tasks  as  a  team.  There  are  infinite  ways  that  a  collective  task 
can  be  performed.  In  addition,  every  commander  has  the  option  of  structuring  his  staff  the 
way  he  wants  to  —  and  new  commanders  often  change  the  unit’s  structure  in  radical  ways. 

Understanding  why  some  teams  perform  better  than  others  requires  that  we  look  at 
the  dynamic  interactions  within  teams  and  the  way  these  interactions  contribute  to  the 
performance  of  the  team.  This  understanding  can  form  the  basis  for  a  descriptive  model  of 
teams  that  can  be  related  to  advanced  team  performance.  Once  a  model  is  developed,  it  can 
be  communicated  to  others  and  can  serve  as  the  basis  for  developing  tools  for  assessing 
teams.  These  assessments  can  be  used  as  direct  feedback  for  immediate  training  purposes  as 
»well  as  providing  guidelines  for  future  training. 

This  report  describes  a  Phase  II  SBIR  project  conducted  by  Klein  Associates  under 
Army  Research  Institute  (ARI)  Contract  MDA903-92-C-0098.  The  overall  goal  was  to 
address  the  above  questions  posed  by  ARI.  The  technical  objectives  for  this  project  were 
threefold.  The  first  objective  involved  the  identification/development  of  a  model  of  advanced 
teams.  The  second  was  to  develop  an  assessment  instrument  based  on  this  model.  The  third 
was  to  evaluate  the  utility  of  the  model  and  the  instrument  using  real-world  teams. 

The  work  described  in  this  report  is  a  culmination  of  many  years  of  Klein  Associates 
research  on  a  wide  range  of  teams.  The  current  model  of  advanced  teams  that  we  present 
here  is  the  result  of  a  series  of  projects  focused  specifically  on  improving  our  understanding 
of  the  nature  of  teams  and  the  implications  of  the  dynamic  interactions  of  their  members. 
During  the  Phase  I  of  this  work  for  the  U.S.  Army  Research  Institute,  we  began  looking  at 
teams  from  the  perspective  of  individual  cognition  and  we  identified  a  preliminary  set  of 
observable  behaviors  that  appeared  indicative  to  team  skills  (Thordsen,  Klein,  Wolf,  & 
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Banks,  1993).  In  a  separately  funded  SBIR  from  ARI,  we  furthered  this  line  of  research  by 
producing  the  original  version  of  the  ATDM  model  (Zsambok,  Klein,  Kyne,  &  Klinger, 
1993),  referred  to  here  as  ATDM  1.0.  The  latest  version,  ATDM  2.0,  was  developed 
during  this  Phase  n  project  for  ARI,  and  is  described  in  this  paper.  Development  of  the 
model  has  also  been  influenced  by  our  work  with  helicopter  crews  (Thordsen,  Klein,  & 

Wolf,  1990),  commercial  aviation  crews  (Klein  &  Thordsen,  1991;  Thordsen  &  Calderwood, 
1989),  Army  battalion-,  brigade-,  and  division-level  planning  teams  (Thordsen,  Galushka, 
Klein,  Young,  &  Brezovic,  1990;  Thordsen,  Klein,  Michel,  &  Sullivan,  1988),  planning 
teams  at  the  National  Defense  University  (Calderwood,  &  Thordsen,  1989;  Zsambok,  Klein, 
Kyne,  &  Klinger,  1993),  wildland  incident  command  firefighting  teams  (Taynor,  Klein,  & 
Thordsen,  1987);  and  Department  of  Energy  emergency  operations  teams  (Klein,  Thordsen, 
&  Calderwood,  1989).  For  clarity,  we  will  always  refer  to  the  earlier  ATDM  model  as 
version  1.0  and  to  the  current  model  as  version  2.0. 

The  evolution  of  the  current  ATDM  2.0  model  reflects  the  lessons  we  have  learned 
from  applying  our  work  in  the  varied  settings  mentioned  above.  The  current  model 
incorporates  team  resources,  team  identity,  team  cognition,  and  team  metacognition  into  a 
four-tiered  framework.  The  two  upper  tiers,  the  Team  Components  and  their  Behavioral 
Dimensions,  provide  a  general  description  of  teams  at  a  domain-independent  level.  The  two 
lower  tiers,  specific  behaviors  and  their  anchors,  allow  the  descriptive  model  to  be 
customized  for  applications  with  specific  domains. 

We  divided  the  project  into  four  stages  in  order  to  address  the  technical  objectives. 

In  stage  one,  we  conducted  an  extensive  review  of  the  current  team  literature.  In  stage  two, 
we  developed  the  team  model  and  an  assessment  instrument.  Stage  three  included  a  field  test 
and  an  evaluation  of  the  model  and  assessment  instrument.  And  finally,  we  developed  a 
prototype  package  of  the  model  and  assessment  instrument  to  show  that  the  materials  could 
•be  transferred  to  the  assessment/training  personnel  of  teams.  A  report  was  generated  for 
each  of  the  first  three  stages.  These  reports  are  referenced  under  Militello,  Kyne,  Klein, 
Getchell-Reiter,  and  Thordsen  (1994),  Thordsen,  Klein,  and  Kyne  (1994),  and  Kyne, 
Thordsen,  and  Kaempf  (1994),  respectively  for  the  first  three  stages.  The  package  of 
materials  we  used  to  transfer  the  information  to  the  domain  subject  matter  experts  (SMEs) 
for  the  test  and  evaluation  (stage  three)  constitute  the  deliverable  for  stage  four.  The  four 
stages  will  be  described  briefly  in  the  following  section  but  we  refer  the  reader  to  the  full 
reports  for  additional  description  and  detail. 

Products  Delivered 


Literature  Review 


Our  primary  goal  was  to  create  a  comprehensive  model  of  team  performance  that 
could  assess  teams  in  any  domain.  For  this  stage  of  the  project,  our  specific  objectives  were 
to  review  current  models  of  team  performance  (including  ATDM  1.0),  identify  common 
themes  and  patterns  among  the  models,  and  synthesize  the  models  by  classifying  the  critical 
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elements  of  team  processes  into  a  common  framework  (see  Militello  et  al.,  1994,  for  a 
complete  description  of  the  review). 

The  team  performance  literature  is  generally  divided  into  three  different  fields:  social 
psychology,  military  science,  and  industrial/organizational  psychology.  Because  of  this,  a 
comprehensive  review  of  team  literature  would  be  an  overwhelming  task  under  the  best  of 
conditions,  and  to  impose  any  time  or  financial  constraints  would  make  it  virtually 
impossible.  Our  challenge  was  to  cover  a  wide  range  of  research  and  simultaneously 
complete  the  task  with  the  resources  available.  To  do  this  we  reviewed  211  team  articles 
from  which  we  identified  several  lines  of  research  that  had  integrated  previous  research  and 
addressed  team  performance  conceptually  in  operational  settings.  We  further  focused  our 
attention  on  lines  of  research  that  contained  a  theoretical  model  and  a  corresponding 
assessment  tool.  Working  from  these  assessment  tools,  we  could  investigate  commonalities 
across  models.  These  commonalities  helped  us  identify  factors  to  incorporate  into  a 
comprehensive  model  of  teams. 

Ultimately,  the  literature  review  focused  on  six  lines  of  team  research.  Each 
employed  different  perspectives  in  their  approach  to  modeling  teams.  Morgan,  Glickman, 
Woodard,  Blaiwes,  and  Salas  (1986)  adopted  a  developmental  view  of  teams.  McIntyre  and 
Dickenson  (1992)  used  a  control  theory  metaphor.  Fleishman  and  Zaccaro  (1993)  prepared 
an  extensive  taxonomy  of  team  functions.  Olmstead  (1992,  1993)  worked  out  the  idea  of  an 
organization  as  a  homeostatic  system  that  adapts  to  achieve  equilibrium.  Helmreich  and 
Foushee  (1993)  relied  on  an  interpersonal  view  of  teams.  Zsambok  et  al.  (1992)  relied  on  a 
cognitive  metaphor. 

While  these  models  were  similar  in  many  respects,  they  were  unique  enough  that 
direct  comparisons  were  difficult.  Therefore,  a  two-stage  analysis  was  adopted  for 
’comparison  of  the  models.  The  initial  stage  involved  extracting  the  comprehensive  set  of  all 
the  team  processes  described  by  the  models.  The  second  stage  of  analysis  involved  mapping 
this  comprehensive  set  of  processes  onto  a  common  framework  so  we  could  make 
comparisons.  To  borrow  a  sports  metaphor,  we  put  them  on  the  same  playing  field.  This 
common  field  allowed  us  to  make  several  types  of  comparisons:  which  specific  processes 
were  addressed  by  which  models;  the  types  of  processes  addressed  by  the  different  models 
(e.g.,  communication  processes,  cognitive  processes,  etc.);  and  the  level  of  abstraction  of  the 
processes  each  model  identified  (e.g.,  abstract  construct,  behavioral  dimension,  specific  team 
behavior,  or  behavioral  anchor  level). 

The  review  allowed  us  to  make  several  observations.  First,  the  models  generally 
converged  on  the  processes  they  identified  as  critical  for  team  performance.  Considering  the 
diversity  of  the  six  models  and  the  fact  that  they  were  based  on  very  different  metaphors, 
the  convergence  between  them  was  somewhat  unexpected.  Second,  the  models  varied  in 
their  degree  of  abstraction.  That  is,  some  models  provided  broad  descriptions  of  team 
processes  while  others  were  tailored  for  specific  teams  in  specific  domains.  Third,  while  we 
expected  to  find  variation  in  the  terminology  used  by  the  models  to  describe  similar 
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processes,  there  was  also  variation  in  the  emphasis  the  models  placed  on  types  of  processes. 
For  example,  all  of  the  models,  either  directly  or  indirectly,  addressed  the  process  of 
leadership,  those  based  on  military  teams  placed  more  emphasis  on  it  as  a  process  than  did 
the  models  from  other  domains.  Some  models  emphasized  personalities  more  than  others, 
some  treated  communications  as  processes  while  others  treated  communication  as  a  link 
between  processes.  Decision  making  was  generally  handled  as  the  communication  of  the 
results  of  the  decision-  making  process.  In  contrast,  the  ATDM  1.0  model  segmented 
decision  making  into  specific  sub-processes. 

When  all  the  processes  were  reduced,  they  converged  on  a  common  set  of  four 
components  that  we  loosely  classified  as  team  competency,  team  identity,  team  planning  and 
decision  making,  and  team  self  management.  Team  competency  addresses  factors  that  can  be 
viewed  as  "pre-team."  In  other  words,  the  "raw  materials"  that  are  available  from  which  to 
build  the  team.  These  include  things  such  as  the  skills  and  expertise  levels  that  the 
individuals  possess.  We  considered  these  "pre-team"  since  their  presence  does  not 
guarantee  they  will  be  used  effectively,  or  even  at  all,  by  the  team.  The  team  identity 
grouping  includes  the  "teamwork"  processes  identified  in  the  models,  such  as  knowledge  of 
how  one’s  roles  and  functions  affect  those  of  other  team  members.  Planning  and  decision 
making  captured  the  cognitive  processes  described  in  the  models,  such  as  the  development  of 
team  situation  assessment.  Finally,  all  team  supervisory  processes,  such  as  time 
management,  were  included  in  the  team  self  management  category. 

The  literature  review  helped  guide  our  revision  of  the  ATDM  1.0  model.  One 
finding  in  particular  caused  us  to  make  a  major  modification.  All  of  the  models  emphasized 
team  leadership  and  team  resources,  except  ATDM  1.0,  which  addressed  them  only 
indirectly.  Based  on  this  observation,  ATDM  2.0  would  need  to  be  expanded  to  cover  these 
processes  more  directly. 

’Model  Development 

Our  goal  was  for  the  ATDM  2.0  model  to  cover  a  full  range  of  team  settings 
including  planning  and  execution  for  tactical  and  strategic,  novice  and  experienced  teams. 

As  a  result,  there  are  four  features  of  ATDM  2.0  that  differentiate  it  from  the  other  models 
of  team  performance  (e.g.,  Fleishman  &  Zaccaro,  1993;  Helmreich  &  Foushee,  1993; 
McIntyre  &  Dickenson,  1992;  Morgan  et  al.  1986;  Olmstead,  1992,  1993;  and  Zsambok  et 
al.,  1992).  First,  it  is  a  comprehensive  model  designed  for  identifying  and  describing  the 
key  elements  that  contribute  to  advanced  team  performance.  Second,  ATDM  2.0  is  guided 
by  a  metaphor  that  allows  one  to  view  a  team  in  much  the  same  manner  as  you  would  view 
any  intelligent  entity.  Third,  the  model  specifically  addresses  the  cognitive  functions  of 
teams.  Many  of  the  models  describe  the  individuals,  resources,  and  lines  of  communication, 
but  few  wrestle  with  the  team’s  decision-making  and  problem-solving  processes.  Fourth, 
ATDM  2.0  is  designed  to  be  both  general  and  specific.  We  felt  it  was  important  that  the 
model  provide  a  general  description  of  teams  and  simultaneously  be  capable  of  being  tailored 
to  specific  teams  in  specific  domains.  Each  model  emphasizes  at  least  one  of  these  features, 
but  ATDM  2.0  is  the  only  model  that  emphasizes  all  four. 
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Figure  1:  A  Model  of  Advanced  Team  Decision  Making,  version  2.0. 


To  help  us  accomplish  the  objective  of  developing  a  model  that  was  both  general  and 
specific  at  the  same  time,  we  organized  the  ATDM  2.0  model  along  four  tiers.  The  two 
higher  tiers  (Team  Components  and  Behavioral  Dimensions)  provide  a  general  description  of 
teams  and  the  factors  that  contribute  to  advanced  team  decision  making  and  performance. 
This  general  model  of  teams,  derived  from  the  two  upper  tiers,  is  described  immediately 
below  and  diagramed  in  Figure  1.  The  two  lower  tiers  (Specific  Behaviors  and  Behavioral 
Anchors)  allow  us  to  specify  observable  team  behaviors  for  specific  domains.  The  Specific 
Behaviors  and  Behavioral  Anchors  are  discussed  in  more  detail  later  but  cannot  be  included 
in  Figure  1  since  they  vary  depending  upon  the  domain  of  interest. 

Both  versions  of  the  ATDM  describe  teams  as  intelligent  entities,  capable  of  learning 
and  adapting,  making  decisions,  reacting  to  surprises,  creating  plans  and  executing  them, 
i.e.,  teams  that  can  demonstrate  cognitive  capabilities.  The  upper  tier  of  ATDM  2.0 
recognizes  four  components  in  describing  a  team:  Team  Resources,  Team  Identity,  Team 
Cognition,  and  Team  Metacognition  (see  Fig.  1).  Team  Resources  involves  recognizing 
what  resources  are  available  to  the  team.  For  an  individual,  these  capabilities  might  involve 
personality  dynamics,  intelligence,  and  experience  base.  For  a  team,  the  cognitive  function 
rests  partially  on  the  individual  capabilities  of  the  team  members  and  partially  on  the  ability 
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of  the  team  to  recognize  and  transition  these  individual  resources  into  resources  serving  the 
entire  team.  Team  Identity  is  the  extent  to  which  individuals  identify  themselves  as  part  of 
the  team.  Erikson  (1963,  1968)  used  identity  to  refer  to  an  individual’s  establishment  of  a 
stable  personal  identity.  One  aspect  involved  selecting  a  role  and  knowing  that  others  could 
clearly  identify  this  role;  that,  in  turn,  led  to  an  increased  sense  of  confidence  and  purpose. 
Teams  also  need  to  develop  a  sense  of  identity,  clarify  the  various  roles  and  functions,  and 
develop  confidence  and  purpose.  Decentering  is  a  key  element  of  Team  Identity  (Flavell, 
1977;  Piaget  &  Inhelder,  1969).  For  a  young  child,  identity  emerges  by  differentiating  self 
from  the  external  world.  For  a  team,  identity  rests  on  its  members  to  associate  themselves 
with  the  output  of  the  team,  rather  than  with  their  own  accomplishments.  Team  Cognition 
addresses  the  cognitive  maturation  of  the  team  to  make  decisions  and  solve  problems.  Many 
of  the  "higher  mental  processes"  (Bernstein,  Roy,  Srull,  &  Wickens,  1991),  such  as 
reasoning  and  problem  solving,  would  be  included  under  cognition.  For  both  individuals  and 
teams,  decision  making  and  problem  solving,  rely  on  processes  for  clarifying  intent,  forming 
situation  awareness,  managing  uncertainty,  and  focusing  attention  effectively.  Team 
Metacognition  is  the  team’s  ability  to  observe  itself  and  adapt.  For  individuals, 
metacognition  refers  to  the  knowledge  and  understanding  they  possess  about  their  cognitive 
activities  and  problem  solving  and  their  efforts  to  regulate  their  ongoing  information 
processing  (Borkowski,  1982;  Brown,  1975;  Flavell,  1980;  Santrock,  1983).  For  the 
individual,  experience  leads  to  metacognitive  skills  for  taking  ones’  own  capabilities  into 
account.  Similarly,  teams  must  learn  metacognitive  or  self-management  skills  to  adapt  to 
their  own  limitations  and  to  the  task  requirements. 

......  These  four  components  constitute  the  highest  tier  of  ATDM  2.0  and  provide  a  broad, 

descriptive  model  of  teams.  Just  below  the  Team  Components  level  is  the  Behavioral 
Dimensions  tier.  Behavioral  Dimensions  are  generic  team  behaviors  that  can  be  viewed  as 
the  critical  dimensions  along  which  each  of  the  Team  Components  vary.  This  variation  is 
critical  in  distinguishing  good  and  poor  team  performance,  opening  the  doors  for  assessment 
'and  training.  While  the  Behavioral  Dimensions  are  less  abstract  than  the  Team  Components, 
they  are  still  domain  independent. 

The  Team  Components  and  Behavioral  Dimensions  described  above  are  general 
enough  that  they  can  be  used  to  describe  any  team,  regardless  of  domain.  To  tailor  the 
model  for  teams  within  particular  domains,  Specific  Behaviors  for  the  Behavioral  Dimensions 
and  Anchors  for  these  Specific  Behaviors  must  be  identified.  These  constitute  the  two  lower 
tiers  of  ATDM  2.0.  This  structure  allows  the  overarching  model  of  teams,  outlined  at  the 
Component  and  Dimension  levels  (see  Fig.  1),  to  be  focused  on  specific  domains.  For  a 
detailed  description  of  all  levels  of  the  model  the  reader  is  referred  to  Thordsen,  Klein,  and 
Kyne  (1994). 
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Development  and  Evaluation 


The  goal  for  this  stage  of  the  project  was  to  draft  a  team  assessment  instrument  based 
on  the  ATDM  2.0  model  and  to  evaluate  the  utility  of  the  instrument  in  a  field  setting.  The 
objectives,  methods,  and  results  of  this  initial  evaluation  study  are  briefly  summarized  below. 
The  reader  is  referred  to  Kyne  et  al.  (1994)  for  a  full  description  of  the  field  test. 

Four  specific  objectives  were  identified  for  this  stage  of  the  project.  First,  we  wanted 
to  test  the  ATDM  2.0  model’s  usefulness  for  describing  team  performance.  We  chose  urban 
firefighting  as  our  test  domain.  Our  second  objective  was  to  gauge  the  amount  of  effort 
required  to  identify  markers  at  the  Specific  Behavior  level.  Third,  we  wanted  to  test  the 
effectiveness  of  the  instrument  for  helping  an  observer  identify  relevant  team  performance 
dimensions  and  rate  the  quality  of  performance.  The  fourth  objective  was  to  evaluate  the 
acceptance  of  the  model  and  instrument  by  domain  experts. 

We  began  this  stage  of  the  project  by  identifying  specific  behavioral  markers  for  the 
fireground  domain.  Earlier  we  had  developed  an  instrument  for  military  battalion-level  battle 
staffs  after  interviewing  two  experienced  battalion  commanders.  This  military  version  of  the 
assessment  instrument  served  as  the  template  for  the  instrument  we  would  use  on  the  para¬ 
military  fireground.  The  military  form  of  the  assessment  instrument  was  modified  for  fire 
fighting  during  a  pilot  study,  at  which  time  the  specific  behavioral  markers  for  the  fireground 
domain  were  identified.  We  found  that  the  specific  behavioral  markers  translated  easily  from 
military  to  para-military.  In  the  pilot  study,  two  researchers  spent  six  days  with  the  fire 
department  of  a  large  U.S.  city.  Using  a  draft  measurement  instrument,  we  accompanied 
eight  Battalion  Chiefs  (BCs)  on  20  incident  calls.  During  this  time,  we  observed  firefighting 
teams  in  action  and  interviewed  the  BCs  and  other  members  of  the  team  both  during  and 
after  incidents.  We  determined  the  validity  of  the  instrument’s  markers  and  whether  it  could 
be  used  easily  in  this  setting.  Based  on  feedback  from  team  members  during  the  pilot  study, 
’we  further  revised  the  instrument  to  the  form  that  we  used  during  the  full  field  test. 

Our  objectives  for  the  field  study  were  to  determine  the  reliability,  useability,  and 
acceptance  of  the  instrument  within  the  constraints  of  an  operational  setting.  Two 
researchers  accompanied  10  BCs  and  one  company  commander  to  36  incidents.  The 
incidents  were  response  calls  to  a  variety  of  emergencies  including  fires,  assist  police,  and 
automobile  accidents.  Some  of  the  teams  were  comprised  of  one  chief  and  a  single  response 
unit  (e.g.,  one  rescue  unit)  while  at  other  times  the  teams  were  made  up  of  multiple 
companies  and,  occasionally,  more  than  one  battalion. 

Our  researchers  responded  to  calls  with  the  BCs.  We  remained  with  the  BC 
throughout  the  incident.  During  each  incident,  we  queried  the  BC  and  his  aide,  monitored 
radio  communications,  observed  team  members’  actions,  and  took  comprehensive  notes. 

After  each  incident,  we  interviewed  team  members  (including  the  BC)  and  completed  a 
rating  form.  The  ratings  were  arrived  at  independently. 
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In  addition,  we  asked  five  of  the  BCs  to  complete  a  rating  form  for  each  incident  in 
which  they  participated  to  assess  whether  the  BCs  viewed  the  incidents  and  applied  the 
assessment  instrument  in  the  same  way  as  the  researchers.  We  assessed  inter-rater  reliability 
between  the  two  researchers  and  between  the  BC  and  the  researchers. 

The  results  indicated  that  the  instrument  derived  from  ATDM  2.0  captured  the  team 
processes  that  were  required  for  successful  performance  on  the  urban  fireground.  The 
observers  and  domain  SMEs  noted  examples  of  each  of  ATDM  2.0’s  behavioral  dimensions 
as  they  assessed  their  teams.  While  not  every  dimension  was  present  in  every  event,  each 
dimension  was  critical  during  at  least  one  of  the  incidents. 

The  specific  behavioral  markers  were  readily  recognized  and  identified  by  outside 
observers  and  by  BCs.  Inter-rater  reliability  for  making  the  categorizations  "Present"  or 
"Not  Observed"  was  assessed  among  researchers  and  BCs  as  well  as  between  researchers. 
Among  researchers  and  BCs  reliability  was  high  (81%  or  above)  for  the  dimensions  of 
member  resources,  leader  resources,  nonhuman  resources,  procedures,  roles  and  functions, 
engaging,  and  assessing  the  situation.  Reliability  was  medium  (61%  to  80%)  for  time 
horizon,  range  of  factors,  and  monitoring  and  adjusting.  Reliability  was  low  (60%  or  below) 
for  compensating,  envisioning  goals,  managing  uncertainty,  and  time  management.  For  29 
of  the  37  total  disagreements  between  the  BCs  and  the  researchers,  the  BCs  indicated  that 
they  observed  a  dimension  that  the  researchers  did  not.  Several  possible  explanations  for 
these  types  of  disagreements  come  to  mind.  First,  it  may  be  that  the  researchers  and  the 
BCs  did  not  share  a  common  understanding  of  the  dimensions.  Second,  it  may  be  that  the 
researchers  and  the  BCs  applied  the  dimensional  definitions  differently.  And  third,  the  BCs, 
being  highly  experienced  in  the  domain,  may  have  been  able  to  notice  cues  and  behaviors 
that  were  not  evident  to  the  researchers. 

•  Our  last  objective  for  this  stage  was  to  get  some  indication  of  whether  the  BCs  would 

accept  the  model/instrument  and  whether  they  would  find  it  helpful.  We  found  their 
acceptance  high.  In  many  cases,  the  BCs  found  the  concepts  reflected  their  own  thinking 
about  team  performance  in  the  domain  and  they  were  able  to  put  these  concepts  into  the 
language  of  the  firefighter.  Interviews  revealed  that  many  BCs  believed  they  had  a  need  to 
improve  their  team  training,  and  that  the  model  and  measurement  instrument  provided  them 
the  tools  to  focus  this  training  on  critical  team  processes. 

Handoff  Package  of  Materials  for  Domain  Users 

Development  of  a  model  of  advanced  teams  does  not  guarantee  that  it  can  be 
packaged  so  that  domain  users  can  take  advantage  of  this  information.  One  of  our  objectives 
was  that  subject  matter  experts  be  able  to  make  use  of  the  model  and  assessment  instrument 
we  developed.  Therefore,  it  was  important  that  we  demonstrate  that  we  could  convey  this 
information  to  potential  users,  such  as  trainers  or  commanders.  In  the  domain  of 
firefighting,  the  potential  users  are  the  fireground  battalion  chiefs.  The  approach  we  adopted 
was  to  develop  a  package  of  materials  that  would  be  handed  off  to  the  chiefs.  This  handoff 
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package  included  a  description  of  the  ATDM  2.0  model,  a  domain-specific  version  of  an 
assessment  tool  derived  from  the  model,  and  a  few  other  associated  materials  such  as  coding 
sheets  and  instructions.  The  handoff  package  was  designed  to  help  BCs  understand  the 
specific  behaviors  critical  to  advanced  team  decision  making  and  performance.  To  be 
successful,  the  handoff  package  needed  to  help  the  BCs  focus  their  attention  on  specific 
behaviors  during  task  execution  in  a  way  what  provided  them  with  information  to  guide 
intervention.  These  interventions  could  take  the  form  of  on-line  adjustments  to  the  team 
during  an  incident,  as  well  as,  identifying  training  needs  that  could  be  addressed  post¬ 
incident. 

Our  goal  was  to  design  the  handoff  package  so  it  would  allow  an  independent 
observer  or  a  knowledgeable  team  member  to  assess  and  document  how  a  team  performed 
during  a  specific  incident.  The  actual  assessment  instrument  requires  the  observer  to  provide 
subjective  ratings  of  team  performance  at  the  level  of  behavior  dimensions  specified  in  the 
ATDM  2.0  model.  The  observer  completes  the  ratings  either  as  the  incident  evolves,  if  time 
permits,  or  after  the  incident  when  the  observer  has  time  to  reflect  on  what  occurred.  In  a 
military  exercise,  the  observer  controllers  would  be  likely  users  of  the  instrument.  Since 
there  are  no  observer  controllers  in  the  firefighting  domain,  the  BCs  who  were  actively 
involved  in  the  incident  were  the  users. 

The  assessment  instrument  included  in  the  handoff  package  was  divided  into  two 
sections.  The  first  was  a  single  sheet  on  which  the  observer  records  his/her  ratings  for  a 
single  incident.  It  permitted  information  to  be  recorded  on  the  Behavioral  Dimensions.  The 
observer  provides  two  types  of  information  for  each  dimension.  First,  they  check  whether 
behavior  indicative  of  the  each  Dimension  was  present  or  not  present  during  the  incident. 

The  second  type  of  information  was  a  subjective  rating  of  the  quality  of  team  performance 
for  those  Dimensions  that  are  present  during  the  incident.  These  scales  are  five-point  rating 
•scales  with  a  scale  value  of  one  anchored  to  "Very  Ineffective,"  and  a  scale  value  of  five 
anchored  to  "Highly  Effective. "  The  observers  use  their  experience  and  knowledge  of  the 
ATDM  2.0  model  to  arrive  at  these  ratings. 

Additional  materials  included  in  the  package  were  intended  to  provide  support  for  the 
observer  as  s/he  observes  team  performance  and  completes  the  rating  scales.  The  section 
serves  as  a  quick  reference  guide  for  the  behavioral  dimensions  of  the  ATDM  2.0  model. 
Grouped  by  component,  each  behavioral  dimension  is  allotted  one  page  that  is  tabbed  for 
easy  access.  This  page  provides  a  definition  of  the  dimension  with  descriptions  of  indicators 
of  effective  performance  and  space  for  the  observer  to  make  notes.  The  observer  can  refer 
to  these  descriptions  before,  during,  or  after  an  incident.  This  facilitates  reliability  among 
observers  and  for  one  observer  across  different  incidents.  The  complete  set  of  handoff 
materials  we  provided  to  the  fireground  commanders  are  attached. 
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Discussion  and  Applications 


Version  1.0  of  the  ATDM  model  has  been  successfully  applied  in  several  settings 
including  strategic  geopolitical,  military  operational  logistics,  and  emergency  operations 
teams.  The  current  model  (version  2.0)  has  successfully  undergone  an  initial  field  test  in  the 
firefighting  domain.  During  this  10-day  field  application,  two  researchers  independently 
rated  each  incident  along  the  dimensions  of  ATDM  2.0.  In  addition,  16  incidents  were  rated 
by  the  chiefs  who  were  the  fireground  commanders  for  each  of  these  incidents.  The  average 
inter-rater  agreement  among  researchers  was  83%  while  the  average  agreement  between  the 
researchers  and  the  chiefs  was  71%.  The  reader  is  referred  to  Kyne  et  al.  (1994)  for  a  more 
detailed  examination  of  this  field  application. 

All  components  and  dimensions  of  the  current  model  have  been  observed  and 
identified  as  important  in  at  least  one  domain  during  the  field  applications  of  the  two  versions 
of  the  ATDM  model.  Still,  while  it  appears  that  all  components  and  dimensions  merit  being 
part  of  the  descriptive,  comprehensive  model,  not  all  are  equally  critical  in  all  domains. 

This  may  allow  the  ATDM  2.0  model,  when  applied  to  a  specific  domain,  to  be  abbreviated 
to  include  only  the  dimensions  that  describe  areas  of  particular  vulnerability  for  that  domain. 
For  initial  testing,  abridged  field  versions  of  the  model  would  have  to  be  developed  on  a 
domain-by-domain  basis,  but  heuristics  might  eventually  be  identified  that  could  be  applied 
across  similar  domains.  For  example,  in  long  standing,  stable  teams,  the  dimensions  of 
Member  Resources  and  Roles  and  Functions  might  be  omitted  but  need  to  be  retained  for 
teams  whose  memberships  frequently  change.  So  while  the  comprehensive  model  needs  to 
include  components  and  dimensions  identified  across  all  teams,  fielded  versions  probably  can 
be  reduced  to  make  them  easier  to  use  and  more  relevant  for  particular  domains. 

How  applicable  is  the  ATDM  2.0  model?  We  feel  the  model  and  field  instruments 
’derived  from  the  model  can  be  instrumental  in  improving  team  assessment,  team  training, 
and  organizational  development. 

With  ATDM  2.0  tailored  for  a  particular  domain,  it  becomes  a  tool  that  can  be 
employed  to  assess  a  team’s  decision  making  and  performance  —  to  identify  the  strengths 
and  weaknesses  of  a  team  along  the  four  components  and  the  13  dimensions  of  the  model. 
This  assessment  can  be  used  to  evaluate  a  team  in  comparison  to  a  standard  or  relative  to 
other  teams  within  the  same  domain. 

The  knowledge  derived  from  the  assessment  of  the  team  about  its  strengths  and 
limitations  can  also  be  used  for  training  purposes.  Areas  where  performance  is  assessed  as 
weak  are  prime  targets  for  training.  This  knowledge  can  focus  training  where  it  is  needed, 
rather  than  letting  a  team  practice  what  it  is  already  good  at  and  thus  wasting  valuable,  but 
often  limited,  training  resources.  An  additional  area  where  use  of  the  ATDM  2.0  model  may 
contribute  to  training  was  mentioned  by  several  fire  department  battalion  chiefs  who  applied 
the  tool  to  their  teams.  They  said  that  by  using  the  model  to  assess  their  teams,  they 
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simultaneously  developed  a  better  understanding  of  the  dynamics  and  vulnerabilities  of  teams 
and  of  team  performance  in  general. 

One  other  area  where  the  ATOM  2.0  model  can  be  applied  is  in  the  area  of 
organizational  development.  If  the  critical  components  and  dimensions  necessary  for  a  team 
to  perform  well  are  understood,  this  information  can  be  used  in  developing  an  appropriate 
organizational  structure  for  a  particular  team,  in  a  particular  domain,  with  particular 
objectives,  responsibilities,  and  tasks.  An  understanding  of  a  comprehensive  model  of  teams, 
such  as  the  one  presented  here  can  provide  a  framework  from  which  specific  issues  and 
concerns  can  be  identified  and  addressed.  Some  of  these  questions  might  be  as  follows: 

How  can  we  structure  the  team  to  make  sure  the  team  members  know  the  strengths  and 
limitations  of  the  other  team  members,  so  we  take  full  advantage  of  these  resources?  In  the 
context  within  which  our  team  operates,  what  can  we  do  to  make  sure  we  elicit  different 
perspectives  about  our  situation  and  ultimately  converge  on  a  common  assessment?  What 
steps  will  we  need  to  take  to  achieve  a  shared  understanding  of  our  objectives?  Are  there 
procedures  we  can  implement  that  will  help  us,  if  necessary,  communicate  changes  in  our 
situation  assessment  and  goals  to  all  the  members  of  the  team?  This  list  is  not  intended  to 
capture  all  the  issues  that  could  be  considered,  however  it  does  provide  a  glimpse  of  how  a 
comprehensive  model  of  teams  can  contribute  to  the  organizational  development  of  a  team. 

A  framework  for  looking  at  team  structure  and  the  performance  implications  of  this  structure 
provides  an  organization  the  opportunity  to  fine-tune  its  teams  to  achieve  more  effective 
performance  that  is  in  line  with  the  overall  goals. 

While  the  inter-rater  agreement  data  and  the  Battalion  Chiefs’  observations  and 
comments  of  version  2.0  of  the  ATOM  model  are  very  encouraging,  we  are  anxious  to 
extend  the  testing  and  evaluation  in  several  directions.  These  include  additional  reliability 
and  validity  testing,  further  study  of  the  generalizability  of  the  descriptive  (upper  two  tiers) 
’portion  of  the  model,  closer  examination  of  the  psychometric  issues  involved  in  adapting  the 
model  to  specific  domains  for  the  application  of  more  rigorous  psychometrics.  One 
additional  area  we  would  like  to  investigate  is  whether  the  model  can  be  used  to  differentiate 
the  level  of  development  of  a  team  along  the  novice-advanced  continuum. 

Summary 

! 

ATOM  2.0  is  a  comprehensive  description  of  teams  that  incorporates  elements  of 
resources,  identity,  cognition,  and  metacognition.  In  addition,  the  model  is  designed  to  both 
provide  a  general  description  of  teams  and  to  be  tailored  for  specific  teams.  While  all  of  the 
individual  behaviors  have  been  addressed  in  some  fashion  in  other  team  models,  our 
objective  was  to  incorporate  them  into  one  model.  The  current  version  has  been  successfully 
applied  with  advanced  execution  teams  (firefighting).  The  previous  version,  which  has  many 
structural  similarities,  has  been  used  successfully  with  novice  teams  (emergency  operation 
centers)  and  strategic,  planning  teams  (the  Industrial  College  for  the  Armed  Forces).  These 
initial  field  applications  have  been  encouraging  although  more  study  is  needed  for  several 
reasons.  First,  to  more  solidly  establish  reliability,  validity,  and  generalizability  of  the 
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model.  Second,  to  identify  psychometric  principles  necessary  for  tailoring  the  general, 
descriptive  model  to  particular  domains.  And  finally,  to  thoroughly  explore  the  ATDM  2.0 
model’s  potential  contribution  to  the  tracking,  assessment,  and  evaluation  of  a  team’s 
development. 
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